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ANALYZING IMAGES TO DETERMINE THIRD PARTY PRODUCT 
MATERIALS CORRESPONDING TO THE ANALYZED IMAGES 
CROSS flEFERENCE TO RELATED APPLICATIONS 

Reference is made to commonly assigned U.S. Patent Application 
5 Serial No. 09/576,558, the disclosure of which is incorporated herein by reference. 

FIELD OF THE INVENTION 
The present invention relates to analyzing one or more images of a 
user to determine the likelihood of user interest in materials that can be sent to the 
user. 

1 0 BACKGROUND OF THE INVENTION 

Photographic imaging service applications have been extended to 
include digital imaging technology. For Internet-based photographic service 
applications, a consumer is provided a means for displaying his/her digital images 
with the digital images residing on a remote computer. Typical scenarios for these 

1 5 Intemet-based digital imaging apphcations include the viewing thumbnail 

versions of a collection of digital images, selection of a particular digital image for 
viewing, enhancement, and/or printing. While there exist many different methods 
for an Intemet-based photographic service provider to receive payment for the 
service rendered, many have chosen to display advertisement messages on the 

20 consumer's display screen and collect payment not from the consumer, but from 
an advertisement chent. At present, it is possible for the photographic service 
provider to perform directed advertisement if prior knowledge of the consumer in 
the form of a consxmier profile is available. However, if no prior knowledge of 
the consumer is available, directed advertising is not possible. Furthermore, the 

25 consumer profile may not be up-to-date. Moreover, the profile may not account 
for some facets of a consumer's buying habits. If an employee of the photographic 
service provider were to view the consumer's photographs, the employee could 
make inteUigent decisions as to which advertisement client would most likely 
desire directed advertisement to the particular consumer. Aside from issues of 

30 privacy of consumer photographs, the process of humans observing photographs 
and making directed advertising decisions is too costly for consideration. 
Research has shown that unrelated directed advertisements are often considered 
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nuisance by the consumer while directed advertisement which relate to the 
interests of the consumer are considered desirable. 

Digital imaging algorithms have for a long while been devised to 
analyze the content of digital images. In particular, the methods disclosed by 
5 CuUen et al. in U.S. Patent 5,933,823, Ravela et al in U.S. Patent 5,987,456, and 
De Bonet et al. in U.S. Patent 5,819,288 analyze digital images. In these digital 
imaging apphcations a database of digital images is maintained. For each digital 
image in the database a set of image features, expressed in mathematical form, are 
calculated. A query digital image is selected, usually initiated from the user of the 

1 0 digital imaging application, and compared to the digital images in the database. 
The same set of image features is calculated for the query digital image. A 
comparison between the calculated image features for the query digital image and 
the database digital images is performed and yields an image similarity value for 
each of the database digital images as a measure of overall similarity. The image 

1 5 similarity values are analyzed and the digital images with the highest image 
similarity values are displayed for the user. 

While these digital image query apphcations are capable of 
analyzing digital images, none of the above mentioned disclosed methods relate 
the content of a set of consumer digital images to the likelihood of an 

20 advertisement chent's desire to direct advertisement material to that particular 
consumer. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a digital imaging 
algorithm, which can make intelhgent directed advertising decisions by analyzing 
25 the image content of consumer digital images. 

It is another object of the present invention to make use of a 
content of a user's image(s) to determine the likelihood that materials would be of 
interest to the user. Such materials can include products or services promotional 
materials. 

This object is achieved by a method of analyzing an image 
provided by a user to determine the likelihood of user interest in materials related 
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to products of third parties and sending such materials for display or printing for 
the user, comprising: 

a) scamiing a hard copy of an image provided by a user to 
provide a digital image and sending such image to a memory location; 
5 b) automatically analyzing the scanned digital image to 

determine the likelihood that materials related to products will be of interest to the 
user by recognizing features which relate to the product of users, such features 
being selected from the group consisting of product trademarks, product trade 
dress, and other products which are related to the third party products; and 
10 c) selecting one or more items of product materials based on 

their likehhood of interest to the user and sending them to the user for display or 
printing. 

ADVANTAGES 

It is an advantage of the present invention that it provides an 
1 5 advertiser or other purveyor of information with the opportunity to automatically 
make inteUigent directed advertising decisions by analyzing the image content of 
consumer digital images. 

It is another advantage of the present invention that by making use 
of a content of a user's image(s), the likelihood can efficiently and effectively be 
20 determined that materials would be of interest to the user. It is a feature of the 
invention that such materials can include products or service promotional 
materials. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a pictorial diagram showing a display of images and 
25 promotional material produced by the present invention; 

FIG. 2A is a block diagram showing communications network of 
connected computers suitable for practicing the present invention; 

FIG. 2B is a block diagram showing the functional blocks of a 
kiosk suitable for use in the present invention; 
30 FIG. 3 is a block diagram showing two computers connected via a 

commimications network including a digital image processor and general control 
processor suitable for practicing the present invention; 
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FIG. 4 is a block diagram showing computer, which includes a 
digital image processor; 

FIG. 5 is a block diagram showing the details of the general control 

processor; 

5 FIG. 6 is a block diagram showing the details of the digital image 

processor; 

FIG. 7 is a block diagram showing the steps image feature 
representation phase; 

FIG. 8 is a block diagram showing the steps of computing the 
10 representation of a digital image in terms of its perceptually significant colors; 

FIG. 9 is a block diagram showing the steps of the second method 
of computing the representation of a digital image in terms of its perceptually 
significant colors; 

FIG. 10 is a block diagram showing the steps of computing the 
1 5 representation of a digital image in terms of its perceptually significant textures; 

FIG. 1 1 is a block diagram showing the conceptual index structure 
used to organize a database of digital images and metadata; and 

FIG, 12 is a block diagram showing the steps of computing the 
image similarity value based on perceptually significant colors and/or textures. 
20 DETAILED DESCRIPTION OF THE INVENTION 

In the following description, a preferred embodiment of the present 
invention will be described as a software program. Those skilled in the art will 
readily recognize that the equivalent of such software may also be constructed in 
hardware. Because image manipulation algorithms and systems are well known, 
25 the present description will be directed in particular to algorithms and systems 

forming part of, or cooperating more directly with, the method in accordance with 
the present invention. Other aspects of such algorithms and systems, and 
hardware and/or software for producing and otherwise processing the image 
signals involved therewith, not specifically shown or described herein may be 
30 selected form such systems, algorithms, components, and elements known in the 
art. Given the description as set forth in the following specification, all software 
implementation thereof is conventional and within the ordinary skill in such arts. 
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Internet-Based Digital Imaging System Application 

The present invention may be implemented with multiple 
computers connected via a communications network. A communications network 
of multiple computers is illustrated in FIG. 2 A. Such a network of connected 
5 computers provides a means of sending and receiving information between any 
two or more connected computers. A commxmications network may include 
physical connections from one computer to another such as can be achieved with a 
conventional communications phone line. It is also possible for the 
communications network to include non-physically connected communications 

10 lines such as can be achieved with microwave communications links, radio 

communications links, coaxial cable television communications links, fiber optic 
communication links, or cellular telephone conmiunications links. Thus, the 
present invention may be practiced with any of the communications systems 
mentioned above, but is not limited solely to these systems since the present 

1 5 invention relies on exchange of information not the means of achieving the 
exchange of information. 

An image capable computer 100 is any device capable of executing 
a series of computational instructions which includes the manipulation of digital 
image data. Although fundamentally any image capable computer may have the 

20 capability to perform any computational instructions, the image capable 

computers illustrated in FIG. 2A have been given specialized names to emphasize 
the special purpose functions they perform. Some examples of image capable 
computers are a connection computer 120, retail computer 130, wholesale 
computer 140, personal computer 150, mobile computer 160, and kiosk 170 which 

25 includes a computer 300 (See FIG. 2B). The present invention may be practiced 
with, but is not limited to, all of the above examples of image capable computers. 
It should also be considered that a given computer may be configured to perform 
more than one function illustrated in FIG. 2 A. However, for the purposes of 
discussion of the present invention, the specialized functions will be referred to as 

30 separate entities even though the physical implementation of the computer 
hardware may not always be separate. 
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The comection computer 120 shown in FIG. 2 A refers to a 
specialized computer device which has been configured for the purposes of 
connecting other computers to the communications computer network 110. Such 
a computer may be the commimications connection for the other types of 
5 computers shown - a retail computer 130, a wholesale computer 140, a personal 
computer 150, a mobile computer 160, and a kiosk computer 170. The main 
function performed by a connection computer 120 is the facilitation of transfer or 
exchange of information rather than the processing of information. A single 
connection computer may be connected to many other computers. As illustrated 

10 in FIG. 2 A, both connection computers shown are coimected to multiple other 

computers and to each other. The arrows in the diagram of FIG. 2 A represent the 
exchange of information which, in general, is bi-directional that is, the flow of 
information may be achieved in either direction. Since information may flow 
from computer to computer for the computers, a network of connected computers 

15 is formed. Computers not shown in diagram of FIG. 2 A might be connected 

symbolically via the communications computer network 110. Thus, any computer 
connected to another computer which is connected to the communications 
network is also connected and may exchange information with any other computer 
connected to the network. For the purposes of the discussion of the present 

20 invention the collection of interconnected computers including the 
communications network will be referred to as the Intemet. 

A personal computer 150, a mobile computer 160, and a kiosk 170 
including computer 300 are shown connected to the communications computer 
network 110 via a connection computer 120. These computers have the capability 

25 for the exchange and display of information. In particular, as it relates to the 

present invention, these computers have the ability to, but are not limited to, the 
display of text, graphic, and image information. Such a computer is typically 
coimected to the Intemet with software, which understands a variety of protocols 
and manages the visual display of information. One such combination of display 

30 software and software protocol is a World Wide Web (WWW) browser which 
imderstands Hypertext Markup Language (HTML). Other display software and 
other software protocols exist. The present invention is not limited to a Web 



-7- 

browser processing HTML documents and may be practiced with any 
combination of software, which manages and displays information. 

A personal computer 150 represents a computer, which is often 
operated by a single person at a time. Typical personal computers are installed in 
5 homes and businesses. Individual users may access the hrtemet with a connected 
personal computer. Personal computers may be portable units such as a lap-top 
computer. If a personal computer is connected to the Intemet with a wireless 
connection it may be located almost anywhere. In such a configuration, the 
personal computer may represent a mobile computer 160. Fundamentally, a 

10 mobile computer and personal computer may differ mostly in size and weight. 

A kiosk 170 including computer 300 may be dedicated to a 
commercial task of performing a specialized service. These kiosks are generally 
owned and maintained by businesses and operated primarily by consumers. An 
example of a typical kiosk such as the Kodak Picture Center manufactured by the 

15 Eastman Kodak Company includes a CRT for displaying information, selecting 
service choices, indicating a payment method for the service selection, and a 
printer for printing a receipt of the transaction. Another example of such a kiosk is 
the Picture Maker kiosk manufactured by the Eastman Kodak Company. A 
fimctional block diagram is shown in FIG. 2B which describes the operations and 

20 functions of the Picture Maker kiosk. Turning briefly to FIG. 2B, kiosk 170 

includes a scanner 302, a CRT 304 for displaying information and images scanned 
by the scanner 302, a printer 306 for printing enlarged images or groups of images 
or other information and the computer 300 for running software capable of 
coordinating the operation of the referenced fixnctional blocks. It should be 

25 recognized that the scanner 302 may be a print scanner such as included with the 
Picture Maker kiosk or it could be film scanner such as included in the Create-a- 
Print kiosk also manufactured by the Eastman Kodak Company. It should also be 
noted that printer 306 may be a dye sublimation thermal printer such as included 
in the Picture Maker kiosk or it could be any other type of printer such as an ink 

30 jet printer or silver halide printer. Continuing with FIG. 2B, computer 300 may 
send images that are scanned by the scanner 302 via the communications 
computer network 1 10 to the wholesale computer 140 for processing according to 
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the present invention using the computing power of wholesale computer 140. 
Wholesale computer 140 subsequently sends the results back to computer 300 via 
the communications computer network 110. The processing results which have 
recognized product trademarks, product trade dress, and other products which are 
5 related to the third party products may include materials to be printed by printer 
306 or an electronic form for the user to input information that can be stored in an 
electronic database of any computer coimected to the commvmications computer 
network 110. 

In a typical situation, a user will present an image to be scanned to 

10 the operator of the kiosk 170. Alternatively, the kiosk 170 may include a self- 
operated system. In either case, software in the computer 300 asks the user a 
number of questions. For example, the computer will solicit the user's interest in 
receiving distribution materials as a part of the sequence of using kiosk 170. A 
hard copy of the image can be made by processing the digital image and using a 

15 printer such as a thermal printer. While this print is being made, other operations 
performed by the computer 300 can simultaneously be worked on, such as the 
soliciting of information from the user. The information supplied by the user can 
be entered into a database either in computer 300, connecting computer 120, or 
wholesale computer 140 shown in FIG. 2 A. Moreover, the user may also furnish 

20 information concerning materials that he or she desires, which may or may not 
relate to the image being scanned. 

Returning to FIG. 2 A, a retail computer 130 represents a computer, 
which may also be dedicated to a commercial task of performing a specialized 
service set in a retail business. These computers are generally owned and 

25 maintained by the retail business and operated either by consumers or store 

personnel. T3^ical retail computers may include a variety of devices connected. 

Referring to FIG. 2 A, a wholesale computer 140 represents a 
computer, which may also be dedicated to a commercial task of performing a 
specialized service. These computers are generally dedicated to high volume 

30 processing tasks of large amounts of data. For example, a wholesale computer 
may contain specialized hardware that enables particularly fast computation of 
image processing functions. The main difference between a retail computer 130 
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and a wholesale computer 140 is in the emphasis or level of user or operator 
assistance required for the software running on these computers. The retail 
computer 130 emphasizes the human interaction with the software while the 
wholesale computer 140 emphasizes the automatic mode of the software. Since 
5 the wholesale computer 140 in general is computationally faster than the retail 

computer 130, images to be processed may be sent from the retail computer 130 to 
the wholesale computer 140, processed on the wholesale computer 140, and sent 
back via the conamunications computer network 1 10 to the retail computer 130. 

It is assumed that all of the above mentioned computers may have 

10 the capabiUty to store the computational instructions, or software in a variety of 
means which include, but are not limited to, random access memory (RAM), read 
only memory (ROM), or some form of off-line storage means such as magnetic or 
optical storage devices. 

Description of an Image and a Digital Image 

15 An image can refer to any form of visual information in recorded 

or displayed form. Examples of recorded images may include, but are not limited 
to, a photographic film negative, a photographic shde film, a motion picture film, 
and a photographic print. Displayed forms of images may include, but are not 
limited to, visual presentations made on electronic displays such as CRT monitors, 

20 LCD panels, electrolimiinescent devices, and LASER projection systems. 

A digital image is comprised of one or more digital image 
channels. Each digital image channel is comprised of a two-dimensional array of 
pixels. Each pixel value relates to the amount of hght received by the image 
capture device 10 corresponding to the geometrical domain of the pixel. For color 

25 imaging applications a digital image will typically consist of red, green, and blue 
digital image channels. Although the preferred embodiment of the present 
invention is practiced with digital images produced with a capture device 10, the 
present invention is not limited to pixel data relating to photographs. For example, 
graphic or other synthetic data may be merged with photographically captured 

30 pixel data and still be considered a digital image. Other configurations are also 
practiced, e.g. cyan, magenta, and yellow digital image channels. For 
monochrome applications, the digital image consists of one digital image channel. 
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Motion imaging applications can be thought of as a time sequence of digital 
images. Those skilled in the art will recognize that the present invention may be 
applied to, but is not limited to, a digital image for any of the above mentioned 
applications. Although the present invention describes a digital image channel as 
a two dimensional array of pixel values arranged by rows and columns, those 
skilled in the art will recognize that the present invention can be applied to mosaic 
(non rectilinear) arrays with equal effect. 

Description of an Image Capable Computer 
The present invention may be implemented in a combination of 
computer hardware software as shown in FIG. 3 within an image capable 
computer 100. Although the preferred embodiment of the present invention is 
most logical to place in a retail computer 130 or wholesale computer 140, both of 
these devices are image capable computers 100. The following description relates 
to a digital imaging system as a subset of the components of an image capable 
computer 100 which includes an image capture device 10, a digital image 
processor 20, an image output device 30, and a general control processor 40. The 
system may include a monitor device 42 such as a computer console or paper 
printer. The system may also include an input device control for an operator such 
as a keyboard and or mouse pointer 44. Multiple image capture devices 10 may 
be connected to the digital image processor 20 as the present invention may be 
used with digital images derived from a variety of imaging devices. For example, 
FIG. 3 may represent a digital photofmishing system where the image capture 
device 10 is a conventional photographic film camera for capturing a scene on 
color negative or reversal film and a film scanner device for scanning the 
developed image on the film and producing a digital image. The digital image 
processor 20 provides the means for processing the digital images to produce 
pleasing looking images on the intended output device or media or altematively, 
for analyzing the content of processed digital images. Although a single image 
output devices 30 is shown, the present invention may be used in conjunction with 
a variety of image output devices 30 which may include a digital photographic 
printer and soft copy display. The digital image processor 20 may process the 
digital image to adjust the overall brightness, tone scale, image structure etc. of 
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the digital image in a manner such that a pleasing looking image is produced by 
an image output device 30. Those skilled in the art will recognize that the present 
invention is not hmited to just these mentioned image processing modules. 

The general control processor 40 shown in FIG. 3 may store the 
present invention as a computer program stored in a computer readable storage 
medium, which may comprise, for example: magnetic storage media such as a 
magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such 
as an optical disc, optical tape, or machine readable bar code; solid state electronic 
storage devices such as random access memory (RAM), or read only memory 
(ROM). The associated computer program implementation of the present 
invention may also be stored on any other physical device or medium employed to 
store a computer program indicated by offline memory device 46. Before 
describing the present invention, it facilitates understanding to note that the 
present invention is preferably utihzed on any well known computer system, such 
as a personal computer 150. 

It should also be noted that the present invention implemented in a 
combination of software and/or hardware is not hmited to devices, which are 
physically connected and/or located within the same physical location. One or 
more of the devices illustrated in FIG. 3 may be located remotely and may be 
connected via a wireless connection. 

Overview of an Image Comparison Digital Imaging System 

The diagram illustrated in FIG. 3 shows two computer systems as 
connected together via a communications computer network 110. The image 
capable computer 100 is connected to the communications computer network 110 
and receives a set of query digital images, which originated from a consximer (a 
person or persons) directly or from kiosk 170 which the consumer is interacting 
with. The retail computer 130 analyzes the set of query digital images for image 
content with respect to a pre-existing database of digital images that may represent 
product trademarks, product trade dress, and other products which are related to 
the third party products. The analysis of the set of query digital images results in 
the selection of one or more distribution items from a database of distribution 
items, which are related to the pre-existing database of digital images. The set of 
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query digital images and the one or more distribution items are transmitted by the 
image capable computer 100 and received by the personal computer 150 or the 
computer 300 within kiosk 170 via the communications computer network 1 10. 
The details of the configurations of how the image capable computer 100 receives 
and/or generates the set of query digital images, how the set of query digital 
images is analyzed, and how the distribution items are selected and transmitted 
will be discussed in more detail below. 

Description of Distribution Items 
The distribution items are received by a personal computer 150 or 
computer 300 of kiosk 170 as shown in FIG. 3. FIG. 1 shows an example display 
screen as presented on a monitor device 42 connected to the personal computer 
150 shown in FIG, 3. Referring to FIG. 1, the items 201, 202, 203, 204, 205, 206, 
and 207 are displayed versions of the digital images included in the set of query 
digital images 220. While the user is browsing the active image, meaning a large 
display of a digital image 201, some of the other individual digital images 
included in the set of query digital images 220 are displayed as thumbnail images, 
or small displays, represented by 202, 203, 204, 205, 206, and 207. The 
distribution items received are presented on the same display screen and are 
represented by distribution items 210, 211, and 212. Examples of distribution 
items from trademark owners that may be practiced with the present invention 
may include, but is not limited to, digital images for promotional purposes, 
graphic or text images which contain promotional messages, combinations of 
digital images merged with graphic or text images, video clips, documents which 
contain promotional messages, HTML web documents, email messages, and 
world-wide-web addresses (Webster). 

The present invention may also be practiced with distribution items 
that do not have a visual representation such as auditory messages in the form of 
audio clips. These audio forms of distribution items the personal computer 150 
can display the information through its intemal speaker. 

An altemative embodiment of the present invention is practiced 
with distribution items that are not in electronic form. For example, such 
distribution items may include, but are not limited to, printed materials or free 
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promotional materials products or services that may be available at the same 
location as kiosk 170. An example of a promotional material or product would be 
a sample of shampoo, soap, or toothpaste being introduced by the owner of 
product trademarks or product trade dress being associated with the query digital 
image. An example of a promotional service would be a coupon for a car wash 
that may be printed directly at kiosk 170. For this embodiment of the present 
invention the image capable computer 100 shown in FIG. 2 A receives a set of 
query digital images from a consumer via kiosk 170 and the communications 
computer network 110. The general control processor 40 shown in FIG. 3 (as a 
component of the image capable computer 100) transmits the results to kiosk 170. 
One or more of the individual digital images included in the set of query digital 
images 220 is printed on the printer 306 of kiosk 170 resulting in a set of digital 
prints. The set of digital prints, along with the distribution items are taken to a 
check out coimter by the consumer. 

The consumer may also have a set of photographic fihn negatives, 
which are physically brought to the retail store without a kiosk. For this case, an 
image capture device 10, as shown in FIG. 3, is a photographic film scanner. The 
set of query digital images is produced by image capture device 10 and received 
by the digital image processor 20. As another scenario, the consumer travels to a 
retail store, which does not have kiosk 170 or other digital imaging equipment. 
The set of photographic film negatives is received by the personnel of the retail 
store. The photographic film negatives are then transmitted to a wholesale 
business, which includes a coiuiected wholesale computer 140 as shown in 
FIG. 2 A. The wholesale computer 140 includes an image capable computer 100 
as shovm in FIG. 3. The set of query digital images and the distribution items are 
handled in similar fashion as described above and are delivered through a delivery 
carrier service, either back to a retail store or directly to the consimier. As still 
another scenario, the consumer has a photographic film scanner connected to a 
personal computer. The digital images produced with the a photographic film 
scanner are received by the retail or wholesale computer electronically. 
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Details of the Comparison Logic 

The digital imaging components of the image capable computer 
100 shown in FIG. 3 is illustrated in more detail in FIG. 4, The digital image 
processor 20 receives multiple sets of database digital images indicated by items 
5 221, 222, and 223. Each set of database digital images contains one or more 
digital images. Also received by the digital image processor 20 is a set of query 
images 220, which contains of one or more digital images. While a set of 
database digital images which may represent product trademarks, product trade 
dress, and other products which are related to the third party products and a set of 

10 query digital images are fundamentally both a collection of digital images, the use 
of these two different sets of digital images is different as practiced by the present 
invention. Alternatively, the set of query images 220 and the multiple sets of 
database images 221, 222, and 223 may have be been initially received by the 
general control processor 40, stored in an offline memory device 46, and then 

1 5 transmitted to the digital image processor 20, 

The individual digital images which make up a set of database 
digital images are related images which have been chosen to be representative of a 
particular theme. Digital images relating to an ocean/beach resort may be 
collected to form a set of database images representing a ocean/beach resort 

20 theme. For example, the digital images in a set of database digital images may 
include images of the beach with sky and clouds, tropical trees, images with 
sailboats, or images with swimming pools. The important aspect of a set of 
database digital images is that the set of images as a whole represent the particular 
theme. Another example of a particular theme would be automobiles. An 

25 example set of database images might include individual digital images of 
automobiles, groups of automobiles, distant automobiles on a race track, or 
people congregating around a food stand. Here, too, the individual digital images 
constituting a set of database digital images are related which supports a particular 
theme. Another example of a particular theme would be a group including product 

30 trademarks, product trade dress, or other products which are related to the third 
party products. 
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Let the multiple sets of database digital images be represented by E 
with each set of database digital images identified with by an index j. Thus Ej 
refers to the j^^ set of database digital images. Each set of database digital images 
may contain a different number of digital images. Let N represents multiple 
numbers with Nj representing the number indicating the number of digital 
images in the j^^ set of database digital images received by the digital image 
processor 20. The individual digital images of a set of database digital images are 
identified with an index k. Thus the k^^ digital image of the set of database 
digital images is represented symbolically as Fjk. The j^^ set of database digital 
images is represented as 

E^ = { Fji, Fj2, .... FjNj}. (1) 
The individual digital images contained in the set of query digital 
images also relate to a single person, a single family, or group of related people 
and represents a collection of consumer digital images. The set of query digital 
images may vary in number and may have been collected over relatively short 
period of time of minutes or hours or over a relatively long period of time such as 
months or years. The important aspect of the set of query digital images is the 
fact that the individual digital images relate to one or more activities of 
importance to the person or people who are either photographed in the digital 
images or photographed by the person or people. The individual digital images of 
the set of query digital images are identified with by an index i. The set of query 
digital images is represented as 

Q - { Qu Q2, Qm} (2) 
where M represents the number of digital images included in the set of query 
digital images and Qi represents the i^ individual digital image of the set. 

Referring to FIG. 4, the general control processor 40 receives 
multiple sets of database digital images. Although there are three sets shown in 
FIG. 4 (items 221, 222, and 223), the present invention may be practiced with any 
number of sets of database digital images. For practical reasons, the number of 
sets may be Umited to the amount of available computer memory, however, the 
present invention as disclosed does not place any restriction on the number of sets 
of database digital images or the number of digital images contained in a set of 
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database digital images. Similarly, the set of query digital images is not restricted 
in number of individual digital images. 

The digital image processor 20 analyzes the set of query digital 
images with respect to multiple set of database digital images. More specifically, 
the individual digital images of the set of query digital images are analyzed with 
respect to the individual digital images of the multiple sets of database digital 
images. 

The numerical analysis performed by the digital image processor 
20 results in a database similarity table 252, represented by the variable y, which 
is a set of database similarity values, i.e. one database similarity value 
corresponding to each set of database digital images where yj represents the 
database similarity value of the j^^ set of database digital images. The database 
similarity values included in the database similarity table 252 are single numerical 
values, which are an indication of how similar the set of query digital images 220 
are to each set of database digital images. Thus, the database similarity table 252 
contains information which relates the set of query digital images 220 to the 
individual sets of database digital images 221, 222, and 223. 

The general control processor 40 shown in FIG. 4 receives the 
database similarity table 252 and multiple sets of distribution materials indicated 
by items 231, 232, and 233. Each set of distribution material includes one or more 
distribution items as described above. The preferred embodiment of the present 
invention uses one set of distribution materials corresponding to each set of 
database digital images. The general control processor 40 uses the database 
similarity table 252 to determine how to distribute the distribution items included 
in the sets of distribution materials. The database similarity values are assessed 
and the general control processor 40 selects one or more distribution items from 
one or more sets of distribution materials to transmit to the consumer's personal 
computer 150. The details of the how the database similarity values yj are 
calculated is described in more detail below. 

Details of the Selection of Distribution Items for Display or Distribution 
The details of the general control processor 40 shown in FIG. 4 is 
illustrated in more detail in FIG. 5. The database similarity table 252 is received 
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by the distribution controller 280 which analyzes the individual database 
similarity values included in the database similarity table 252. The individual 
database similarity values y j included in the database similarity table 252 are 
sorted in descending order to produce a database ranking table 254 represented by 
the variable R, The indices of the ranking process are saved in a table of numbers. 
This table of number is essentially a look-up-table of the relative similarity of the 
set of query digital images to the individual sets of database digital images. For 
example, if five sets of database digital images are received and the corresponding 
database similarity table 252 is given by 

Y= { 89 45 73 62 91 }, (3) 
the resulting database ranking table 254 is given by 

R - {5 1 3 4 2 }. (4) 
Referring to FIG. 4, the database similarity values yj produced by 
the digital image processor 20 range in numerical value from 0 to 100. Higher 
values represent a greater degree of similarity. The preferred embodiment of the 
present invention selects the set of database digital images with the greatest degree 
of similarity to the set of query digital images 220. Thus the set of database 
digital images with the highest corresponding database similarity value, 
represented by the numerical index Ri, is selected and also selects the set of 
distribution materials corresponding to the selected set of database digital images. 

The display screen on CRT 304 of kiosk 170 can display one or 
more than one distribution items. The selected set of distribution materials may 
have more distribution items than the personal computer can display at one time. 
The preferred embodiment of the present invention cycles through the selected set 
of distribution materials by selecting individual distribution items for transmission 
to the kiosk 170. For example, if the kiosk 170 has the ability of displaying two 
distribution items and the selected set of distribution materials includes ten 
distribution items, the first two distribution items are selected, transmitted, and 
displayed on the CRT 304 of kiosk 170. After a time period of five seconds, the 
next two distribution items, the third and fourth, are selected, transmitted, and 
displayed on the CRT 304 of kiosk 170. This process is repeated until all of the 
individual distribution items included in the selected set of distribution items has 
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been transmitted and displayed at which point the process of cycling through the 
distribution items is repeated. 

Those skilled in the art will recognize that the present invention 
may be used effectively with other configurations of sequencing through a set of 
5 distribution items. For example, once the set of distribution materials is 

determined, the individual distribution items may be selected at random. This 
type of display of information breaks up the monotony for the person viewing the 
display screen. It should also be noted that the time period of displaying 
distribution items is completely up to the digital imaging appUcation designer. 

10 An alternative embodiment of the present invention displays 

distribution items from more than one set of distribution. In this embodiment, 
individual distribution items from two or more sets of distribution materials are 
selected, transmitted, and displayed on the personal computer 150. The two sets 
of database digital images with the two highest associated database similarity 

1 5 values are selected (indicated by indices Rl and R2). If the personal computer has 
the ability to display two distribution items at a time, one distribution item from 
the first set of distribution materials is displayed in a portion of the personal 
computer display while one distribution item from the second set of distribution 
materials is displayed in the other portion. Each of the sets of the distribution 

20 materials is cycled through the respective individual distribution items as 
described above. 

Another alternative embodiment of the present invention displays 
distribution items from one or more sets of distribution materials simultaneously 
and varies the length of time that the distribution items are displayed on the CRT 

25 304 of kiosk 170. In this embodiment, individual distribution items from two or 
more sets of distribution materials are selected, transmitted, and displayed on the 
CRT 304 of kiosk 170. In each portion of the CRT 304 of kiosk 170 devoted to 
displaying a distribution item, a distribution item selected from the set of 
distribution materials with the highest associated database similarity value is 

30 selected (Ri), transmitted, and displayed. These distribution items are displayed 
for eight seconds. After eight seconds have elapsed, a distribution item from the 
set of distribution materials with the next highest associated database similarity 
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value is selected (R2), transmitted, and displayed. These distribution items are 
displayed for three seconds. In this manner, the length of time devoted to the 
display of distribution items is related to the corresponding database similarity 
values. The process continues until all of the sets of distribution materials have 
been selected. 

Those skilled in the art will recognize that the present invention 
may be used effectively with other configurations of varying the length of time 
devoted to the display of distribution items. For example, the set of distribution 
materials may be selected at random with the individual distribution items cycled 
sequentially. Each time a distribution item is selected, the length of time it 
remains on the personal computer display is determined by its associated database 
similarity value. It should also be noted, as above, that the actual length of time 
for displaying a distribution item is completely up to the designer of the digital 
imaging application. 

Calculation of the Database Similarity Values 
The digital image processor 20 as part of an image capable 
computer 100 shown in FIG. 4 is illustrated in more detail in FIG. 6. The image 
similarity calculator 240 receives multiple sets of database digital images 
indicated by items 221, 222, and 223 and a set of query images 220. The image 
similarity calculator 240 performs a numerical analysis of each digital image Qi 
included in the set of query images 220 with each digital image Fjk included in 
each set of database digital images. The numerical analysis performed by the 
image similarity calculator 240 is an image-to-image comparison, i.e. a 
comparison of two digital images, which results in a single numerical value 
termed a image similarity value for each two image comparison. Thus since there 
are M digital images included in the set of query digital images and there are Nj 
digital images in the j^^ set of database digital images, the total number of image- 
to-image comparisons performed Nt is given by 

Nt = M Sj Nj. (5) 
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The image similarity value resulting from the image-to-image comparison of the 
i*^ digital image of the set of query digital images and the k^^ digital image of the 
j*^ set of database digital images is represented by pyk^ 

The image database evaluator 250 receives the image similarity 
5 values from the image similarity calculator 240 and derives a table of database 
similarity values. Each database similarity value is a single numerical value that 
relates to the degree to which set of query digital images is similar to a set of 
database digital images. Recall that the variable yj represents the database 
similarity value corresponding to the j^^ set of database digital images. 
1 0 The database similarity values may be calculated in variety of 

different ways. The preferred embodiment of the present invention uses the 
arithmetic mean of the image similarity values to derive a database similarity 
value. The equation for calculating the yj is given by 

Yj ^ Pijk (6) 

1 5 which represents the average similarity value. 

An alternative embodiment of the present invention uses a subset 
of the image similarity values to calculate a database similarity value. The image 
similarity values pijk (there are M Nj values) corresponding the j^^ set of database 
digital images are ranked in numerical order. An average of only the five highest 

20 image similarity values is used to calculate the database similarity value. Those 
skilled in the art will recognize that the present invention can be practiced with a 
different number of the highest image similarity values. Using only the highest 
image similarity values raises the numerical values of the database similarity 
values. Since only a few digital images contribute to the calculation of the 

25 database similarity value no penalty is placed on the inclusion of a large number 
of digital images for a given set of database digital images. 

Calculation of the Image similarity values 
The present invention can be practiced with any method of 
producing image similarity values. The essential elements of a digital image 

30 processing method required to practice the present invention are the calculation of 
one or more image features for a database of digital images and a query digital 
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image and a method for calculating a single measure of similarity of the calculated 
image features. For example, the present invention could be practiced with 
adapted versions of, but is not limited to, the methods disclosed by CuUen et al. in 
U.S. Patent 5,933,823; Ravela et al. in U.S. Patent 5,987,456; De Bonet in U.S. 
5 Patent 5,819,288; Choo et al. in U.S. Patent 5,832,131; Barber et al in U.S. Patent 
5,579,471; and described by MJ. Swain and D.H. Ballard in ''Color indexing, 
Intl. Journal, of Computer Vision, " Vol 7, No. 1, 1991, pp, 11-32; and G. Pass, et 
al in ''Comparing images using color coherence vectors, " Proceedings ACM 
Multimedia Conf,, 1996 since all of these methods are image similarity methods 

10 based on one or more calculated image features. 

The present invention provides a depictive feature-based image 
comparison system, which consists of two functional phases. In the first phase, 
called the image feature representation phase, every digital image in the multiple 
sets of database images managed by the system is processed to automatically 

15 extract its depictive feature-based. The image feature representation and the 

digital image are stored in a database and a search index is updated to enable the 
image feature representation to participate in future depictive feature-based image 
comparisons. The second phase, called the image comparison phase, is concemed 
with the comparison of the digital images included in the set of query digital 

20 images with the digital images included in the sets of database digital images. 

Note that the image color space can be transformed into any predefined or desired 
color space for archival and retrieval phase. The embodiment details given below 
are applicable to digital images of any color space (e.g., RGB, YCC, HSV, CIE 
color spaces, etc.). Also, digital images can be transformed to a desired 

25 compressed dynamic range in both phases to reduce the computational cost and 
storage requirements. 

Details of the Image Feature Representation Phase 
The key steps of the image feature representation phase are shown 
in FIG. 7. Each input digital image is analyzed to build its representation. A 

30 digital image or a digital image of product trademarks, product trade dress, and 
other products which are related to the third party products can be represented in 
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terms of several different depictive features such as color, texture, and color 
composition. 

Referring to FIG. 7, a depictive feature /is first selected, SIO. In 
the preferred embodiment, either color or texture can be selected. An example of 
a depictive color feature is the specific color of the Kodak logo. The digital image 
is analyzed to identify its perceptually significant components in terms of 
feature/, S20. A depictive feature of a digital image is considered a perceptually 
significant depictive feature if it dominates an observer's perception of the digital 
image. For example, the color red is a perceptually significant color feature in an 
image of a large red flower. The digital hnage is then represented in terms of the 
detected perceptually significant components of the selected feature, S30. The 
input digital image and its representation are then added to the image database, 
S40. 

According to the present invention, color feature-based 
representation of a digital image is in terms of perceptually significant colors 
present in the digital image. The preferred approach to identifying perceptually 
significant colors of a digital image is based on the assumption that significantly 
sized coherently colored regions of a digital image are perceptually significant. 
Therefore, colors of significantly sized coherently colored regions are considered 
to be perceptually significant colors. The preferred embodiment offers two 
different methods for the identification of perceptually significant colors of a 
digital image. One of these methods is selected for setting up a database. The key 
steps of the first approach are shown in FIG. 8. For every input digital image, its 
coherent color histogram is first computed, SI 00. A coherent color histogram of a 
digital image is a function of the form H (c) = number of pixels of color c that 
belong to coherently colored regions. Here c is a valid color in the dynamic range 
of the digital image. A pixel is considered to belong to a coherently colored 
region if its color is equal or similar to the colors of a pre-specified minimum 
nxmiber of neighboring pixels. The present implementation has two definitions of 
coherency: (i) a minimum of two matching or similar neighbors, and (ii) all 
neighbors are matching/similar. The same coherency definition must be used for 
analyzing all digital images in both the image archival and retrieval phases. Two 
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colors are considered equal if all the corresponding channel values are equal. Two 
colors ci and C2 are considered similar if their difference dijf(cu C2) is less than a 
user specified threshold Cr. The preferred value of Ct is in the range of 1 5% to 
20% of the maximum possible value of diffifcj, C2). Several different color 
difference computation methods are possible. In the preferred embodiment one of 
the following three methods for comparing two L-channel colors can be selected 
at the system initialization time: 

(i) Color Cx and Cy are considered similar if \cx - Cy\ < Ct\ 
where denotes the value of the color digital image channel q 
and Ch denotes the pre-specified threshold value for the difference 
of the color digital image channel values. 

(ii) Color Cx and Cy are considered similar if 2;=7,i:W/. - Chf < 
Ct, where q denotes the value of the color digital image 
channel Ch, Wj is the weight of the color digital image channel, 
and Ct denotes the pre-specified threshold value. 

(iii) Color Cx and Cy are considered similar if i^*=/,LW/. \(cx - Ch) \ 
< Ct, where Ck denotes the value of the color digital image 
channel Ch, Wi is the weight of the color digital image channel, 
and Ct denotes the pre-specified threshold value. 

Then the coherent color histogram is analyzed to identify the 
perceptually significant colors, SllO. A color k is considered to a perceptually 
significant color \fH(k) > T. Here Tis a threshold. In the present implementation 
r= 0.5% of total numbers of pixels in the image. The next step is to represent the 
digital image in terms of its perceptually significant colors, S120. Specifically, a 
digital image / is represented by a vector of the form 

Ic = {N, Z, <Ci, Si>} , 0 < i < N+1 . (7) 
Here, N is the nimiber of perceptually significant colors in digital image I, Z = i7 
5^/, Ci is the color value of the i^^ perceptually significant color of digital image I, 
and Si is the ratio of H(Ci) and the total number of pixel in image L 

The key steps of the second method for identifying perceptually 
significant colors of a digital image are shown in FIG. 9. This method is an 
extension of the first method. In this case, steps SI 00 and SllO of the first 
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method are performed to detect perceptually significant colors, S200. The set of 
perceptually significant colors so obtained is considered the initial set of 
perceptually significant colors and it is refined to obtain the set of dominant 
perceptually significant colors. The refinement processed starts with the finding 
5 of connected components composed solely of the pixels of colors belonging to the 
initial set of perceptually significant colors, S210. This is accomplished by 
performing connected component analysis on the input digital image considering 
only the pixels of perceptually significant colors (i.e., considering them as the 
object pixels) and ignoring others (i.e., considering them as the background 

10 pixels). Two neighboring pixels (4 or 8 -neighbors) with perceptually significant 
colors (i.e., colors in the initial set of the perceptually significant colors) are 
considered connected only if they are of matching/similar colors. The connected 
components so obtain are analyzed to determine the set of dominant perceptually 
significant colors, S220. A connected component of size greater than a pre- 

15 specified threshold Ts is considered a dominant perceptually significant segment. 
In the present implementation, Ts = 0.25% of the total number of pixel in the 
digital image. Colors belonging to a dominant perceptually significant segments 
form the set of perceptually significant colors for image feature representation. 
The final step is again to represent the digital image in terms of its perceptually 

20 significant colors, S230. Note that this final set of perceptually significant colors 
is a subset of the initial set of perceptually significant colors. 

Those skilled in art would recognize that several variations of the 
above two color-based image feature representations are possible within the scope 
of this work. For example, one straightforward extension is a combination of the 

25 two representations, where the representation of method 1 is extended by 

qualifying each perceptually significant color by a type which indicates whether 
or not that colors belongs to a dominant perceptually significant segment. 

According to the present invention, texture feature-based 
representation of a digital image is in terms of perceptually significant textures 

30 present in the digital image, random or structured. The preferred approach to 
identifying perceptually significant textures of a digital image is based on the 
assumption that each perceptually significantly texture is composed of large 
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numbers of repetitions of the same color transition(s). Therefore, by identifying 
the frequently occurring color transitions and analyzing their textural properties, 
perceptually significant textures can be extracted and represented. The preferred 
embodiment for the identification of perceptually significant textures of a digital 
image is shown in FIG. 10. For every input digital image, the first step in the 
process is to detect all the color transitions that are present in the image, S300. A 
color transition occurs between a current pixel (c) and its previous pixel (p) if a 
change of color value, dist(c, p), is greater than a predefined threshold th. The 
preferred value of th is in the range of 15% to 20% of the maximum possible 
value of dist(c, p), A pixel where a color transition occurs is referred to as a 
color-transition-pixel. In the present embodiment, one of the following two 
methods for comparing two L-channel colors can be selected to determine the 
occurrence of a change of color value, hence, a color transition: 

(i) The current pixel is identified to be a color-transition-pixel 
if 

\c,cho-p,cho\ > tkcho or \c.chj -p.chj] > th.chj ... (7) 
or |c.chi- p.chil > th.chj 0 < i < L 
where achf represents the color digital image channel value of 
the current pixel, p. chi represents the color channel value of the 
previous pixel, and th.chi represents the predefined difference 
threshold for the color digital image channel. 

(ii) The current pixel is identified to be a color-transition-pixel 
if 

\Y,{c,ch--p,ch,f >th 

where c.chf represents the color digital image channel value of 
the current pixel, p.chf represents the color digital image channel 
value of the previous pixel, and th represents the predefined color 
difference threshold. 

Those skilled in art would recognize that the concept of color 
transition can be defined as gray-level or brightness transition in case of 
monochromatic images. They would also recognize that other color difference 
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metrics can be employed for determining the existence of a color transition within 
the scope of this invention. A digital image is scanned horizontally and vertically 
to identify all color-transition-pixels using one of the above methods. Every 
color-transition-pixel signals a color transition and each color transition is 
represented by the two colors (cl, c2) corresponding to the previous and the 
current pixel color values that form the color transition. The second step in the 
process is to identify all the frequently occurring color transitions, S3 10. Two 
dimensional color transition histograms with cl and c2 being the two dimensions 
are constructed to record the frequency of various color transitions found in the 
previous step. The preferred embodiment of the present invention offers three 
options for constructing and populating the color transition histograms. The first 
option involves the construction of a global color transition histogram, which will 
be populated by all the color transitions found in the image. Finding all the peaks 
in the color transition histogram that also exceeds a predefined minimum 
frequency of occurrence threshold identifies the frequently occurring color 
transitions. The preferred minimum frequency threshold for identifying 
frequently occurring color transitions for global color transition histogram is 
0.25% of the total number of pixels in the digital image. The second option 
involves tessellating the digital image into non-overlapping sections and then 
constructing a set of sectional transition histograms, which are populated by color 
transitions, found in corresponding image sections. In the present embodiment, a 
set of 24 sectional histograms is constructed. Finding all the peaks in all of the 
sectional transition histograms that also exceeds a predefined minimum frequency 
of occurrence threshold identifies the frequently occurring color transitions. The 
preferred minimum frequency threshold for identifying frequently occurring color 
transitions for sectional color transition histogram is 2.5% of the total number of 
pixels in each tessellated section. The final option is a combination of the two 
above-mentioned methods where both the global and the sectional histograms are 
constructed and all the peaks are identified in the above-mentioned manner. 
These peaks represent the most frequently occurring color transitions, which 
correspond to perceptually significant textures in the image. The third step in the 
process is texture property analysis of frequently occurring color transitions to 
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represent perceptually significant textures, S320. For every frequently occurring 
color transition, all the occurrences of this particular color transition in the entire 
digital image are found, and a scale and a gradient value is calculated. In the 
current embodiment, scale is calculated as the distance in terms of pixels, between 
the occurrence of color cl and color c2. Gradient is calculated as tan^(gy/gx) 
where gy and gx are the vertical and horizontal edge information at the color 
transition respectively, calculated using the Sobel operator. Note that other 
techniques for calculating scale and gradient values are possible without 
exceeding the scope of this invention. The calculated scale and gradient values 
for each occurrence are used to construct a scale-gradient histogram. After all the 
occurrences have been accounted, the scale-gradient histogram is used to analyze 
the textural properties of the perceptually significant texture. For random 
textures, the scale gradient histogram is randomly distributed, while for structured 
textures, a significantly sharp mode in scale, gradient, or both can be detected in 
the scale-gradient histogram. For a color transition corresponding to a random 
texture, the scale-gradient histogram is used to compute the scale-gradient mean 
vector and the scale-gradient co-variance matrix. For a color transition 
corresponding to a structured texture, the corresponding histogram mode is used 
to compute the scale-gradient mean vector and the scale-gradient co-variance 
matrix. These properties are used to represent a perceptually significant texture. 
The final step is to represent a digital image in terms of its perceptually significant 
textures, S330. A digital image I is represented by a vector of the form: 

It = {N, Z, < C\, C^, Pi, Mi, Vi, Si,>}, 0 < i < N+1. (9) 
Here iVis the number of dominant perceptually significant textures in image 
/; Z = ZSi, di and C\ are the color value of the frequently occurring color 
transition corresponding to the perceptually significant texture; Pi is the textural 
type of the perceptually significant texture, taking on one of the following 
possible values: random, scale-structured, gradient-structured, or scale-gradient- 
structured; Mi and Vi are the scale-gradient mean vector and scale-gradient 
CO variance matrix of the perceptually significant texture in the set, respectively, 
and Si is the total area coverage of the i^^ perceptually significant texture 
calculated by accumulating all the scale values over all the occurrences of the 
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frequently occurring color transition corresponding to the perceptually 
significant texture. Those skilled in art would recognize that other textural 
properties, or a subset/superset of /r can be employed to represent a perceptually 
significant texture. 

After generating the perceptually significant feature-based image 
feature representation, the next step is to insert the digital image and the 
associated representation into the corresponding database and appropriate index 
structures. Those skilled in art would recognize that the overall database 
organization is dependent on the underlying database/file management system. In 
the present implementation, the digital images reside in the image database. The 
image feature representations (metadata) are also stored in the database, as well as 
in the indexing structures. In addition to the perceptually significant feature 
representations, an image feature representation (metadata) also contains the 
image identifier/locator which act as a reference to the digital image file. The 
image name/id acts as a locator of its representation. Note that in the current 
implementation color and texture representations are organized in separate 
structures, but they share the common digital image. 

The steps outlined in FIG. 7, FIG. 8, FIG. 9, and FIG. 10 are 
repeated for all the digital images in each set of database digital images. This 
represents a database of digital images and metadata that does not need to be 
changed unless a new set of database digital images is to be included. The present 
invention may also be practiced with the many different sets of query digital 
images. However, for each set of query digital images the steps outlined in 
FIG. 7, FIG. 8, FIG. 9, and FIG. 10 are repeated for all the digital images in each 
set of query digital images. A separate database and index structure is maintained 
for each set of query digital images. Therefore, the present invention may be used 
with multiple sets of query digital images and use the same sets of database digital 
images and corresponding database and index structure. 

Image Comparison Phase 
In the image comparison phase, each query digital image in the set 
of query digital images is analyzed separately. An image similarity value is 
computed for the comparison between a query digital image and a database digital 



-29- 

image. This process is repeated for all the query images. Before a similarity 
value is calculated between a query digital image and a database digital image, 
certain constraints must be satisfied by the database image with respect to the 
query image. In the preferred embodiment of the present invention, these 
5 constraints on the perceptually significant features are: (i) minimum niraiber 
perceptually significant query digital image's features that are present in the 
database image's features; (ii) the minimum percentage of the total size of the 
perceptually significant features that are common between the query digital image 
and the database digital image; and (iii) a logical combination of the first two 

10 constraints. To accomplish this, the appropriate index search is first performed to 
select database digital images that contain one or more perceptually significant 
features (or principal perceptually significant features if principal perceptually 
significant feature-based index is employed) of the query digital image. The 
representation of every selected database digital image is then analyzed to 

15 determine if it satisfies the specified size constraints mentioned above, and for 
database digital images that satisfy the constraints, a measure of similarity with 
the query digital image, or image similarity value, is computed. For the database 
digital image which are not selected as part of the above procedure, a image 
similarity value of zero is given since the comparison between the query digital 

20 image and the database digital image did not satisfy the constrains on the 
perceptually significant features. 

The key steps of the example image-based similar image 
retrieval/selection process are shown in FIG. 12, Given a query digital image, its 
desired representation (i.e., color or texture-based) is computed shown in step 

25 S500. Next, the database and the associated index structure is searched to find 
candidate database digital images that can potentially meet the search criteria, 
S510. This is accomplished by searching the index structure to identify digital 
images that contain at least one perceptually significant feature (or principal 
perceptually significant feature if principal perceptually significant feature-based 

30 representation is employed) common with the query digital image (as indicated in 
step S520). For the preferred index structure, this is accomplished by searching 
the index structure for every perceptually significant (or principal perceptually 
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significant) feature^ to find database digital images with feature^ as a 
perceptually significant (or principal perceptually significant) feature. For each 
database digital image that satisfies the search/retrieval constraints, an image 
similarity value is computed in step S530. For database digital images that do not 
satisfy the search/retrieval constraints, an image similarity value of zero is 
assigned. 

For color-based image comparison, the index structure based on 
perceptually significant or principal perceptually significant colors is searched to 
find database digital unages containing in their representation at least one of the 
query digital image's perceptually significant (or principal perceptually 
significant) color. The preferred options for a measure of similarity, the 
calculation of the image similarity value as indicated in step S520, for the color- 
based representations are: 

{a) Diq,d) = f^mm{Sf,Sf) (10) 

(b) D{q,d)^diff(n\n') (11) 

where in N is the number of matching colors of query digital image q and 
database digital image Sf and Sf are the size attribute values for the 
matching color of images q and d, respectively; diff is a normalized distance 
function of the type Ljc\. \ norm for a given x; and £2^ and /3d are the set of size 
attribute values of the corresponding perceptually significant colors of digital 
images q and d. 

For texture-based image comparison, the index structure based on 
perceptually significant (or principal perceptually significant) textures is searched 
to find database digital images that both contain at least one of the query digital 
image's perceptually significant (or principal perceptually significant) textures in 
their representation. The resulting set of candidate database digital images is 
further compared with the query digital image to determine the candidate's 
texture-based similarity to the query image. The preferred measure of similarity 
between the query digital image and the candidate digital image is dependent on 
the similarity of matching, or common, perceptually significant textures, and also 
the total area coverage in the query digital image and the candidate digital image 
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by the matching/common perceptually significant textures. Two perceptually 
significant textures are matching/common if they have matching color values CI, 
C2, and the same texture property value P (random or structured) in their 
representation. In the preferred embodiment, for each matching/common 
perceptually significant texture, the similarity of matching perceptually significant 
texture, the image similarity value, is calculated fi:om the scale-gradient mean 
vector M and the scale-gradient covariance matrix Fusing either Euclidean 
distance or Mahalanobis distance. Note that other distance functions may also be 
used within the scope of the present invention. The overall image similarity score 
between the candidate and the query digital image is determined as the sum of the 
similarity value of all the matching perceptually significant textures multiplied by 
the relative area coverage S of the texture in the digital image. Note that in 
general, the Mahalanobis distance is not a symmetric distance, the distance from 
distribution A to distribution B is different from the distance from distribution B to 
distribution A, In addition, the relative area coverage S is different in the 
candidate and the query digital image. Therefore, two image similarity values 
generally result from the similarity calculation: one from the query to the 
candidate Sq.c. and one from the candidate to the query Sc-q^ The preferred 
embodiment has 5 different options for obtaining one single image similarity 
value. The first two options take either Sg-c or Sc^q as the final image similarity 
value; the third option takes the maximum of Sq.c and Sc-q, the fourth option takes 
the average of 5'^.cand Sc-q, and the fifth option takes the product of Sq.c and Sc-q. 
Other combinatorial methods can also be used without exceeding the scope of the 
present invention. 

The calculation of the image similarity value as described in the 
preferred embodiment of the present invention which includes index structure 
searches and similarity calculation only on the candidate database images can 
result in many query/database digital image comparisons that have a zero value. 
In an altemative embodiment of the present invention, the indexing structure can 
be ehminated. In this embodiment, every database digital image will be selected 
for fiirther evaluation. The method of calculating the image similarity value is 
employed as in the preferred embodiment as indicated by equations (10) and (11). 
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Those skilled in art would recognize that other similarity measures 
could be employed within the scope of the present invention. 

A computer program product may include one or more storage 
medium, for example; magnetic storage media such as magnetic disk (such as a 
5 floppy disk) or magnetic tape; optical storage media such as optical disk, optical 
tape, or machine readable bar code; solid-state electronic storage devices such as 
random access memory (RAM), or read-only memory (ROM); or any other 
physical device or media employed to store a computer program having 
instructions for practicing a method according to the present invention. 
1 0 The invention has been described in detail with particular reference 

to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. 
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PARTS LIST 



10 


image capture device 


20 


digital image processor 


30 


image output device 


40 


general control processor 


42 


monitor device 


46 


offline memory device 


100 


image capable computer 


110 


communications computer network 


120 


connection computer 


130 


retail computer 


140 


wholesale computer 


150 


personal computer 


160 


mobile computer 


170 


kiosk computer 


201 


digital image 


202 


thumbnail image 


203 


thumbnail image 


204 


thumbnail image 


205 


thumbnail image 


206 


thumbnail image 


207 


thumbnail image 


210 


distribution item 


91 1 


Hi ct'HHntirMi ifptn 

U-lblilUULUJli ILCill 


212 


distribution item 


220 


set of query digital images 


221 


set of database digital images 


222 


set of database digital images 


223 


set of database digital images 


231 


set of distribution materials 
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PARTS LIST (con't) 



232 


set of distribution materials 


233 


set of distribution materials 


240 


image similarity calculator 


250 


image database evaluator 


252 


database similarity table 


254 


database ranking table 


280 


distribution controller 


300 


computer 


302 


scanner 


304 


CRT 


306 


printer 


SIO 


software block 


S20 


software block 


S30 


software block 


S40 


software block 


SlOO 


software block 


SllO 


software block 


S120 


software block 


S200 


software block 


S210 


software block 


S220 


software block 


S230 


software block 


S300 


software block 


1 n 
iU 


soirware diock 


S320 


software block 


S330 


software block 


S500 


software block 


S510 


software block 


S520 


software block 


S530 


software block 



WHAT IS CLAIMED IS: 
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1 . A method of analyzing an image provided by a user to 
determine the likehhood of user interest in materials related to products of third 
parties and sending such materials for display or printing for the user, comprising: 

a) scanning a hard copy of an image provided by a user to 
provide a digital image and sending such image to a memory location; 

b) automatically analyzing the scanned digital image to 
determine the likelihood that materials related to products will be of interest to the 
user by recognizing features which relate to the product of users, such features 
being selected from the group consisting of product trademarks, product trade 
dress, and other products which are related to the third party products; and 

c) selecting one or more items of product materials based on 
their likelihood of interest to the user and sending them to the user for display or 
printing. 

2. The method of claim 1 ftirther including the step of printing 
the received user image. 

3. The method of claim 1 wherein the scanning is provided by 
a film scanner or a hard copy printer scanner. 

4. The method of claim 1 further including collecting 
information about the user and storing such information in a database where it can 
subsequently be sent to third parties. 

5. A method of printing an image and analyzing such image to 
determine the likelihood of user interest in materials that can be sent for display or 
printing for the user, comprising: 

a) scanning a hard copy of an image provided by a user to 
provide a digital image and sending such image to a memory location; 
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b) receiving the digital image from the memory location and 
printing such digital image; 

c) automatically analyzing the image to determine the 
likelihood that materials in a set will be of interest to the user; and 

d) automatically selecting one or more items of materials 
based on their likelihood of interest to the user and sending them to the user for 
display or printing. 

6. The method of claim 5 wherein the scanning is provided by 
a film scanner or a hard copy printer scanner. 

7. The method of claim 5 wherein steps a), b), and c) are 
provided in a kiosk, 

8. The method of claim 5 wherein the method is practiced in a 

kiosk. 



9. The method of claim 7 wherein the one or more items of 
materials are printed by the kiosk. 
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ABSTRACT OF THE DISCLOSURE 

A method of analyzing an image provided by a user to determine 
the likeUhood of user interest in materials related to products of third parties and 
sending such materials for display or printing for the user including scanning a 
hard copy of an image provided by a user to provide a digital image and sending 
such image to a memory location, automatically analyzing the scanned digital 
image to determine the likelihood that materials related to products will be of 
interest to the user by recognizing features which relate to the product of users, 
such features being selected from the group consisting of product trademarks, 
product trade dress, and other products which are related to the third party 
products, and selecting one or more items of product materials based on their 
likelihood of interest to the user and sending them to the user for display or 
printing. 
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81670RLO 



As below named inventor, I hereby declare that: 
My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are listed 
below) of the subject matter which is claimed and for which a patent is sought on the invention entitled: 

ANALYZING IMAGES TO DETERMINE THIRD PARTY PRODUCT MATERIALS 
CORRESPONDING TO THE ANALYZED IMAGES 



The specification of which (check only one item below): 
is attached hereto. 



was filed as United States Application Serial No. on and 
was amended on (if applicable). 

was filed as POT international application Number on and was amended under POT Article 1 9 on (if applicable). 



I hereby state that I have reviewed and understand the contents of the above-identified specification, including the claims, as amended by any amendment 
referred to above. 

I acknowledge the duty to disclose to the U.S. Patent & Trademark Office all information known to me to be material to patentability as defined in Title 
37, Code of Federal Regulations, § i .56. 

I hereby claim foreign priority benefits under Title 35, United States Code, § 1 19 of any foreign application(s) for patent or inventor's certificate or of any 
PCT international application(s) designating at least one country other than the United States of America listed below and have also identified below any 
foreign applications (s) for patent or inventor's certificate or any PCT international application(s) designating a least one country other than the United 

States of America filed by me on the same subject matter having a filing date before that of the application(s) of which priority is claimed: 

PRIOR FOREIGN/PCT APPLICATION(S) AND ANY PRIORITY CLAIMS UNDER 35 U.S.C. 119: 



COUNTRY 
(ffPCT, indicate PCT) 



APPLICATION NUMBER 



PRIC^ITY CLAIMED UNDER 35 USC §1 19 



I hereby claim the benefit under Title 35, United States Code, 119 §(e) of any United States provisional application(s) listed below: 



PRIOR PROVISIONAL APPLiCATION(S) AND ANY PRIORITY CLAIMS UNDER 35 U.S.C. §119 (e): 



PROViSICMAL i^PUCATION NUMBER 



I hereby claim the benefit under Title 35, United States Code, § 120 of any prior United States appUcation(s) or PCT international application(s) designating 
the United States of America that is/are Usted below and, insofar as the subject matter of each of the claims of this application is not disclosed in that/those 
prior applications(s) in the manner provided by the first paragraph of Title 35, § 112, I acknowledge the duty to disclose to the U.S. Patent & Trademark 
Office all information known to me to be material to patentability as defined in Title 37, Code of Federal Regulations §1.56, which became available 
between the filing date of the prior appUcation(s) and the national or PCT international filing date of this application: 



PRIOR US APPLICATIONS OR PCT INTERNATIONAL APPLICATIONS DESIGNATING THE U.S FOR BENEFIT UNDER 
35USC§120: ^^^^ 



U.S. APPLICATIONS 



STATUS (Check one) 



U.S. APPLICATION NUMBER 



U.S. FILING DATE 



PATENTED 



PENDING 



ABANDONED 
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PCT FILING DATE 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the attorney(s) and/or 
agent(s) associated with Eastman Kodak Company Customer No> 01333 to prosecute 
this application and transact all business in the Patent and Trademark Office connected 
therewith. 



Send Correspondence to: 


Patent Legal Staff 
Eastman Kodak Company 
343 State Street 
Rochester, NY 14650-2201 


Direct Telephone Calls to: 

Raymond L. Owens 
(716) 477-4653 
FAX: (1\6)A11A6A6 
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FtRST GIVEN NAME 
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CITY 
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STATE OR FOREIGN COUNTRY 
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COUNTRY OF CITIZENSHIP 

USA 
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BUSINESS 
ADDRESS 


BUSINESS ADDRESS 

Eastman Kodak Company 


CITY 

343 State Street, Rochester 


STATE & ZIP CODE (COUNTRY) 
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